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ABSTRACT 

The  existing  spatial  smoothing  (SS)  technique,  although  it  is  effective  in  decorrelating  coherent 
signals,  is  considered  applicable  only  to  uniformly  spaced  linear  arrays  which  are  very  sensitive  to 
the  directions-of-arrival  (DOA’s)  and  can  be  used  to  estimate  azimuth  angles  only.  To  significantly 
improve  the  robustness  of  DOA  estimation  and  of  beamforming  and  to  estimate  both  azimuth  and 
elevation  angles  in  a  3D  multipath  mobile  radio  environment,  we  developed  techniques  for  applying 
SS  to  arrays  of  nonlinear  geometry.  We  found  and  proved  the  necessary  and  sufficient  conditions  on 
an  array  configuration  for  applying  SS.  This  array  must  have  an  orientational  invariance  structure 
with  an  ambiguity  free  center  array,  and  the  number  of  subarrays  must  be  larger  than  or  equal 
to  the  size  of  the  largest  group  of  coherent  signals.  We  also  studied  the  cause  of  ambiguities  in  a 
multipath  environment.  We  found  the  necessary  and  sufficient  conditions  for  a  three-sensor  array 
manifold  to  be  ambiguity  free  and  identified  several  higher  order  ambiguity  situations.  If  an  array 
is  also  central  symmetric,  the  forward/backward  spatial  smoothing  can  be  used  to  improve  the 
resolution.  Finally,  we  expanded  the  application  of  our  technique  not  only  to  MUSIC  and  adaptive 
beamforming  algorithms  but  also  to  ESPRIT  algorithm.  All  the  predicted  results  are  verified  by 
simulations. 


This  work  was  supported  in  part  by  the  NSF  grant  MIP-93-09-506,  the  ONR  grant 
N00014-93-1-0566,  and  MIPS/Watkins  Johnson. 


I  Introduction 


Sensor  array  processing  has  been  a  key  technology  in  radar,  sonar,  communications  and  biomed¬ 
ical  signal  processing.  Recently,  as  the  cellular  communication  technology  advances,  sensor  array 
processing  emerges  as  a  potential  technology  to  improve  the  spectral  efficiency  [1],  [2],  [3]. 

Much  of  the  work  in  array  processing  has  focused  on  methods  for  high-resolution  DOA  estima¬ 
tion  and  optimum  adaptive  beamforming.  These  methods  include  the  well  known  MUSIC  [4]  algo¬ 
rithm  and  ESPRIT  [5]  algorithm  for  DOA  estimation  and  MVDR  and  LCMV  algorithms  [6],  [7],  [8] 
for  beamforming.  However,  an  important  drawback  of  these  techniques  is  the  severe  degradation  of 
the  estimation  accuracy  in  DOA  estimation  [9]  or  signal  cancellation  [10]  in  adaptive  beamforming, 
in  the  presence  of  highly  correlated  or  coherent  signals. 

To  counter  the  deleterious  effects  due  to  some  coherent  signals,  a  pre-processing  scheme  referred 
to  as  spatial  smoothing  (SS)  proposed  by  Evans  et  al.  [11]  and  further  developed  by  Shan  et 
al.  [9],  [12]  has  been  shown  to  be  effective  in  decorrelating  coherent  signals.  However,  such  a 
scheme  is  only  applied  to  uniformly  spaced  linear  arrays.  Linear  arrays  are  known  to  be  limited 
to  estimating  azimuth  angles  within  180°,  and  practically  effective  only  for  signals  from  broadside 
direction.  The  degree  of  SS  using  a  uniformly  spaced  linear  array  is  also  sensitive  to  DOA’s  [13].  As 
a  result,  a  linear  array  is  not  very  effective  in  radar,  sonar,  or  especially  in  cellular  communications 
where  users  can  never  predict  the  incoming  directions  of  the  moving  targets. 

In  the  past  decade,  research  has  been  carried  out  in  developing  algorithms  for  coherent  in¬ 
terference  using  arrays  of  arbitrary  geometry.  In  the  area  of  DOA  estimation,  multidimensional 
subspace  fitting  algorithms  such  as  deterministic  maximum  likelihood  (DML)  [14],  multidimen¬ 
sional  (MD)-MUSIC  [15],  and  recently  proposed  weighted  subspace  fitting  (WSF)  [16],  [17],  are 
effective  in  both  coherent  and  noncoherent  environment  and  can  be  applied  to  arrays  of  arbitrary 
geometry.  However,  all  these  algorithms  involve  some  searching  procedures  used  to  solve  nonlin¬ 
ear  equations.  They  are  computationally  intensive  and  are  not  practical  in  real-time  applications. 
Several  other  techniques  map  the  signal  received  by  the  array  to  a  virtual  array,  but  these  methods 
all  need  approximations  and  have  restricted  applications.  In  the  area  of  narrow-band  adaptive 
beamforming,  The  coherent  interference  suppression  using  null  constraint  with  an  array  of  arbi¬ 
trary  geometry  was  addressed  in  [18].  This  approach  still  requires  pre-estimation  of  arrival  angles 
of  coherent  interferences.  The  SPT-LCMV  beamforming  algorithm  applicable  to  arrays  of  arbi¬ 
trary  geometry  was  considered  in  [19].  This  algorithm  requires  increased  computational  complexity 
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compared  to  LCMV.  Recently,  diversity  combining  [21]  and  blind  adaptive  beamforming  [20]  have 
been  proposed  to  combat  multipath  fading  and  cochannel  interference.  However,  blind  adaptive 
beamformer  has  a  low  convergence  rate,  and  is  only  applied  to  signals  with  constant  modulation. 
Both  techniques  have  limitations  on  tracking  complicated  channels  while  cochannel  interference 
and  multipath  effects  coexist. 

In  this  work,  we  develop  a  general  SS  technique  for  arrays  of  arbitrary  geometry  to  make  MU¬ 
SIC,  ESPRIT  algorithms  and  optimum  adaptive  beamforming  algorithms  operative  in  a  coherent 
interference  environment  and  meanwhile  achieve  robustness  in  performance.  Compared  with  the 
aforementioned  methods  for  arrays  of  arbitrary  geometry,  this  SS  technique  can  be  easily  imple¬ 
mented.  It  does  not  increase  the  computational  complexity  of  either  MUSIC,  ESPRIT,  or  adaptive 
beamforming.  It  allows  us  to  work  on  a  data  domain  [22],  and  thus  enables  us  to  incorporate  the 
recently  developed  URV  [23]  [24]  algorithm  to  DOA  estimation  and  updating  and  enables  us  to 
implement  MVDR  beamforming  algorithm  using  systolic  arrays.  Therefore  it  has  great  potential 
in  mobile  radio  communication  where  coherent  and  cochannel  multipath  interference  is  a  major 
problem.  Also,  it  can  be  used  in  conjunction  with  MUSIC  or  ESPRIT  algorithm  to  provide  an 
initialization  for  the  WSF  method  to  get  a  more  accurate  DOA  estimation  [17]. 

Specifically,  we  discovered  and  proved  the  necessary  and  sufficient  conditions  on  an  array  geom¬ 
etry  for  applying  SS.  They  are:  (1)  such  an  array  must  have  an  orientational  invariance  structure; 
(2)  its  center  array  has  an  ambiguity  free  array  manifold;  and  (3)  the  number  of  subarrays  is  larger 
than  or  equal  to  the  largest  number  of  mutually  coherent  signals.  By  working  on  a  smoothed 
data  matrix  obtained  from  SS,  we  can  use  MUSIC  and  optimum  adaptive  beamformers  effectively 
in  a  coherent  interference  environment.  To  further  increase  efficiency  and  estimation  resolution, 
we  found  that  the  forward/backward  spatial  smoothing  [25]  (FBSS),  when  applied  to  a  nonlinear 
array  of  central  symmetry,  can  reduce  the  number  of  sensors  required  and  improve  the  estimation 
resolution  for  closely  spaced  incoming  signals.  Finally  we  expand  the  application  of  our  results  to 
ESPRIT. 

In  all  the  papers  cited  above  that  dealt  with  DOA  estimation  with  arrays  of  arbitrary  geometry, 
ambiguity  free  array  manifolds  were  assumed.  In  [4]  Schmidt  discovered  and  defined  the  rank-n 
ambiguity  in  an  array  manifold.  In  [28],  Lo  and  Marple  proved  the  conditions  for  a  rank-2  ambiguity. 
In  [26]  ambiguities  of  linear  arrays  were  studied.  However,  constructing  an  nonlinear  array  free  of  up 
to  rank-k  ambiguities  using  only  ( k  +  1)  sensors  remains  a  challenging  problem  [27].  In  this  paper, 
we  report  a  more  thorough  study  on  this  issue.  We  proved  the  necessary  and  sufficient  conditions 
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for  a  three-sensor  array  manifold  to  be  ambiguity  free.  We  then  identified  several  situations,  for 
higher  order  sensor  array  manifolds,  in  which  ambiguity  may  arise.  Thus  we  get  corresponding 
necessary  conditions  to  design  ambiguity  free  center  arrays  and  subarrays. 

This  paper  is  divided  into  six  sections.  In  section  II,  we  prove  the  necessary  and  sufficient 
conditions  on  an  array  of  nonlinear  geometry  for  applying  SS,  and  consider  the  FBSS  technique  for 
applications  in  nonlinear  arrays.  In  section  III,  we  study  the  cause  of  ambiguities  in  a  multipath 
signal  environment.  In  section  IV,  we  present  some  practical  considerations  and  simulation  results. 
In  section  V,  we  expand  our  results  to  ESPRIT.  Section  VI  concludes  our  work. 

II  SS  for  Array  of  Arbitrary  Geometry 

We  first  assume  that  all  the  sensors  in  an  array  discussed  in  this  paper  are  omnidirectional  and 
identical.  Consider  an  array  of  p  sensors.  Let  d  narrow-band  signals  with  additive  white  Gaussian 
noise  impinge  on  the  array  at  incident  angles  9i,  ■  ■  • ,  Q&.  The  array  output  covariance  matrix  has 
the  form  [22] 


R  —  E{r(t)rH(t))  =  ARsAH  +  a2I,  (1) 

where  r (t)  is  the  received  signal  vector  by  the  array  at  time  t,  A  is  a  p  x  d  steering  matrix  and 
a2  is  the  variance  of  the  white  Gaussian  noise.  When  there  are  coherent  interferences,  the  signal 
covariance  matrix  Rs  is  no  longer  full  rank.  Therefore,  all  the  high  resolution  DOA  estimation 
methods  based  on  eigendecomposition  and  all  the  adaptive  beamforming  algorithms  which  assume 
that  interfering  signals  are  not  fully  correlated  with  the  desired  signals  fail  to  operate  effectively. 

In  the  case  of  a  uniformly  spaced  linear  array,  with  a  sensor  spacing  A,  the  SS  algorithm  [9]  [12] 
can  be  applied  to  achieve  the  nonsingularity  of  the  modified  covariance  matrix  of  the  signals. 
This  technique  begins  by  dividing  a  uniformly  spaced  linear  array  of  L  sensors  into  K  overlapping 
subarrays  of  size  p,  with  sensors  {1,  •  ■  •  ,p}  forming  the  first  subarray,  and  sensors  {2,  •  •  •  ,p  +  1} 
forming  the  second  subarray,  etc.  It  was  shown  that  [9] 

Ak  =  (2) 

where  Ak,  k  =  1,  •  •  • ,  K,  is  a  p  x  d  steering  matrix  consisting  of  steering  vectors  associated  with  the 
kth  subarray,  and  E^>  denotes  the  kth  power  of  a  d  x  d  diagonal  matrix  E. 
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The  spatially  smoothed  covariance  matrix  is  defined  as  the  average  of  the  subarray  covariances: 

1  K 

R  =  Rk  =  A\RaA\  +  cr2/, ,  (3) 

n  fc= 1 

where  Rk  is  the  covariance  matrix  associated  with  the  kth  subarray,  Rs  is  the  modified  covariance 
matrix  of  the  signals,  and  has  been  proved  [9]  to  be  full  rank  when  K  >  d.  The  signals  are  thus 
progressively  decorrelated  [13].  However,  linear  arrays  have  limitations  in  the  domain  of  estimable 
DOA’s.  It  has  been  shown  in  [29]  that  Rs  can  be  decomposed  as  follows: 

s.  =  - CCH  (4) 

where  C  —  PAT  with  P  =  diag(pi,p2,  •  •  •  ,Pd),  and 

1  1  •••  1 

e-j27rf  sin(fli)  e~J2nT  sin(^2)  ...  e~j2nf  sin(Od) 

A=  g— j47Ty  sin(0i)  e-j47T^sin(e2)  ...  e~j4ir^  sin(0d) 

e-j2Kn§s'm(ei)  e~j2Kir$  sin(02)  ...  e-j2Kirf  sin(0d) 

When  incoming  signals  are  closely  spaced,  the  columns  of  both  A  and  Ai  become  almost  linearly 
dependent  [29].  The  dependency  increases  drastically  when  some  of  sin(6i),i  =  1,  •  •  • ,  d  approach  1 
for  DOA’s  near  90°.  As  a  result,  the  performance  of  a  linear  array  deteriorates  quickly  when  some 
of  DOA’s  approach  90°.  The  highly  directional  sensitivity  of  the  linear  array  causes  the  lack  of 
performance  robustness  to  the  DOA’s  and  limits  the  domain  of  estimable  angles  to  azimuth  angles 
from  broadside  direction  of  the  array.  The  lack  of  performance  robustness  of  a  linear  array  is  even 
more  severe  when  SS  technique  is  applied,  because  in  the  smoothed  covariance  matrix,  not  only 
the  steering  matrix  A\,  but  also  A  is  ill-conditioned  in  the  situation  described  above.  A  general 
SS  technique  that  is  robust  and  can  be  applied  to  directionally  independent  arrays  is  thus  more 
desirable. 

A  Orientational  Invariance  Structure 

It  is  apparent  that  the  mapping  relation  between  Ak  and  Aj  is  the  key  to  successful  application  of 
the  SS  technique.  In  general,  we  can  divide  an  arbitrary  array  into  K  subarrays  which  may  overlap. 
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There  is  not  always  a  steering  matrix  A  to  map  all  the  steering  matrix  Ak,  for  k  =  1,  •  •  • ,  K  to 
A.  In  this  section,  we  will  develop  necessary  and  sufficient  conditions  on  array  geometries  for 
implementing  the  general  SS.  First,  we  give  the  following  lemmas. 

Lemma  1  For  steering  matrices  A  and  B,  given  by  A  =  [a(0i),  ■  •  a(0d)]  and  B  —  [b(#i),  •  • 

b(#ti)],  there  exists  a  mapping  relation  B  =  AC  if  and  only  if  C  is  a  diagonal  matrix. 

Lemma  2  For  K  steering  matrices  Ai ,  A2,  ■  ■  ■ ,  Ax,  each  Ai  can  be  mapped  to  a  steering  matrix  B 
if  and  only  if  there  exists  a  mapping  relation  Aj  =  AiCij  for  any  i  and  j. 

The  proofs  of  both  lemmas  are  given  in  the  Appendix. 

Consider  an  array  that  is  divided  into  K  subarrays.  Suppose  Ai  and  Aj  are  the  steering 
matrices  associated  with  the  ith  and  the  jth  subarrays,  and  there  are  d  signals  with  incoming 
angles  6\,  •  •  • ,  Od-  Ai  can  be  written  as 

Aj  =  [ai(0i),  ai(02),  •  •  • ,  ai(0d)],  (6) 

where  a J{0k)  =  [e~^il^k\ e~^i2^k\ •  ■  ■ , e~^iv^k\k  =  1  is  the  steering  vector  associated 

with  the  ith  subarray,  and  4>u{0k),  l  G  {1,  •  •  •  ,p},  is  the  phase  delay  of  the  kth  signal  at  the  ith 
sensor  of  the  ith  subarray  from  the  first  sensor  of  the  first  subarray.  We  refer  to  the  sensor  of  an 
array  associated  with  the  Ith  row  of  a  steering  matrix  of  the  array  as  the  Ith  sensor  of  the  array. 

Let  A iji,  1  <  l  <  p,  represent  the  distance  between  the  ith  sensor  in  the  ith  subarray  and  the 
ith  sensor  in  the  jr'th  subarray.  Let  represent  the  angle  of  the  line  on  which  these  two  sensors 
are  located.  If  the  ith  and  the  jth  subarrays  are  identical  and  have  the  same  orientation,  i.e.  all 
A for  i  =  1,  •  •  •  ,p  are  equal  and  all  i  =  1,  •  •  •  ,p  are  equal,  then  the  phase  delay  of  a  signal 
with  an  incoming  angle  9k  from  each  sensor  in  the  ith  subarray  to  the  corresponding  sensor  in  the 
jth  subarray  is  the  same  according  to  the  far  field  assumption.  We  denote  this  phase  delay  by 
$ij{9k)-  For  any  i  G  {1,  •  ■  •  ,p},  we  have 

®ij{8k)  =  <Pji{9k )  -  <t>u{0k)  =  27r~ -6k  +  |),  (7) 

then  Aj  =  AiCij,  where  Cl3  is  a  diagonal  matrix  with  the  mth  diagonal  element  e^]^l^0rn\  The 
identical  and  orientational  invariance  properties  between  two  subarrays  guarantee  a  mapping  rela¬ 
tion  between  their  steering  matrices. 
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On  the  other  hand,  if  Aj  =  A{C,  by  Lemma  1,  C  should  be  a  diagonal  matrix  and  can  be 
represented  by  C  =  diag{cn  (9i),  022(62),  •  •  • ,  cdd(6d)}.  It  requires  that 

e-iM°>)Ckk(Ok)  =  e-W*)  for  l  =  (8) 

which  can  be  simplified  to 

<t>jl(6k)  ~  4>n(6k)  =  $'ij(Ok)  +  2t rn,  for  l  =  1,  -  ■  •  ,p  (9) 

where  n  can  be  any  integer.  The  relation  in  (9)  holds  for  all  9k  in  [0, 360)  only  if  Ay  1  =  Ay2  = 
•  •  •  =  A ijp  and  /3yi  =  =  •  •  •  =  Sijp,  i.e.  the  ith  and  the  jth  subarrays  must  be  identical  and 

have  the  same  orientation.  Thus,  we  have  Lemma  3: 

Lemma  3  Suppose  Ai  and  Aj  are  steering  matrices  associated  with  the  ith  and  the  jth  subarrays. 
The  sensors  in  each  subarray  are  numbered  in  the  same  sequence.  There  exists  a  mapping  relation 
Aj  =  AiCij  if  and  only  if  the  ith  and  the  jth  subarrays  are  identical  and  have  the  same  orientation. 

From  Lemmas  2  and  3,  we  have: 

Theorem  1  Suppose  an  array  can  be  divided  into  K  subarrays,  each  having  apxd  steering  matrix 
Ai,  i  =  1, 2,  •  •  • ,  K.  All  A\,  A2,  •  •  • ,  Ak  can  be  mapped  to  a  p  x  d  steering  matrix  B  by  Ai  =  BDi 
if  and  only  if  all  these  subarrays  are  identical  and  have  the  same  orientation. 

We  call  the  array  structure  held  by  an  array  satisfying  conditions  in  Theorem  1  the  orientational 
invariance  structure.  A  more  rigorous  definition  is  given  as  follows: 

Definition  1  (Orientational  Invariance  Structure)  An  array  has  an  orientational  invariance 
structure  if  it  can  be  divided  into  subarrays  that  are  identical  and  have  the  same  orientation. 

For  an  array  with  orientational  invariance  structure,  we  can  consider  each  subarray  as  one 
element  located  at  its  first  sensor.  Then  all  these  elements  form  a  center  array.  A  more  rigorous 
definition  for  center  array  is  given  as  follows: 

Definition  2  (Center  Array)  If  an  array  with  orientational  invariance  structure  is  divided  into 
subarrays  (which  can  have  overlap),  then  the  collection  of  all  the  first  sensors  of  these  subarrays 
form  a  center  array. 
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B  Necessary  and  Sufficient  Conditions 

Suppose  an  array  has  an  orientational  invariance  structure.  Moreover,  its  center  array  has  an 
ambiguity  free  structure  and  the  number  of  subarrays  is  larger  than  or  equal  to  the  largest  number 
of  mutually  coherent  signals.  The  p  x  d  steering  matrices  Ai,  A2,  •  •  ■  ,Ak  are  associated  with 
the  subarrays  1,2,  •  •  •  ,K,  respectively,  and  dk  is  the  distance  between  the  first  sensor  in  the  first 
subarray  and  the  first  sensor  in  the  kth  subarray.  The  angle  (3k  represents  the  direction  of  the  line 
on  which  the  first  sensor  in  the  first  subarray  and  the  first  sensor  in  the  kth  subarray  are  located 
(see  Fig.l).  We  have 

Ak  =  A\Dk,  k  =  2,---,K  (10) 


where 


Dk 


sin03*.— 0i  +  f) 


s-J^X^sinOSfc-^+f ) 


.-inr1  sin(/3*-0d+f) 


(11) 


The  covariance  matrix  of  the  kth  subarray  is  thus  given  by 


Rk  =  AxDkRsD%A f  +  a2 1,  (12) 


where  Rs  is  the  covariance  matrix  of  the  source.  The  spatially  smoothed  covariance  matrix  is 
defined  as  the  average  of  the  subarray  covariances 


1  K 

R  =  ^  £  Rk  =  AiRaAi  +  a2/,  (13) 


where  Rs  is  the  modified  covariance  matrix  of  the  signal  given  by 


Rs  =  ^DkRsD%.  (14) 

K  k= 1 


We  will  show  in  the  following  that  Rs  is  nonsingular.  First,  Rs  can  be  written  as 
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1  D 

K^s 

I 

1  p 

Kns 

D? 

Rs  =  [ID2  Dk ] 

to 

Co 

i _ 

to 

1 _ 

Let  C  denote  the  Hermitian  square  root  of  ^Rs,  i-e. 

CCH  =  hRs' 

It  follows  that 

Rs  =  GGh 

where  G  is  a  d  x  Kd  block  matrix  given  by 

G  =  [C  D2C  ■■■  DkC ]. 


(15) 


(16) 


(17) 


(18) 


Clearly,  the  rank  of  Rs  is  equal  to  the  rank  of  G.  Suppose  there  are  q  groups  of  signals  in  d  incoming 
signals,  with  lq,  i  —  1,  •  •  ■ ,  q,  correlated  signals  in  each  group.  Rs  must  be  a  block  diagonal  matrix 
with  block  size  Zj,  i  =  1,  •  •  •  ,q.  We  can  thus  get  a  corresponding  block  diagonal  matrix  C.  Recall 
that  the  rank  of  a  matrix  is  unchanged  after  a  change  of  its  columns.  By  grouping  columns  of 
similar  elements,  we  can  verify  that 


P(G)  =  p 


cipbi  ■■■  ci^jbi 


ch,  ibh  ciuhhh 


Cd-lq+l,d-lq+lbd-lg+l  ■  ■  ■  Cd-lq+l,dbd-lq+l 


Cd,d-lq+\bd 


Cd,dbd 


(19) 


where  p  is  a  rank  operator,  is  the  *jth  element  of  matrix  C,  and  bj  (i  =  1,  •  •  • ,  d)  is  the  1  x  K 
row  vector  given  by 


bj  =  [1  e 


-j  sin(P 2  -0i  +  f )  p -  j  sm»(03  -  0, + f ) 


•  ■  e 


-3- 


lsin(pK-e,+%) 


(20) 
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Each  row  of  matrix  C  has  at  least  one  nonzero  element  because  the  energy  of  each  signal  is  nonzero. 
It  is  observed  that  bj  is  the  transpose  of  the  steering  vector  associated  with  the  center  array.  Since 
the  center  array  is  assumed  to  have  ambiguity  free  array  manifold,  when  K  >  max{/i,  •  •  • , lq}, 
all  the  b  vectors  associated  with  all  the  signals  within  a  group  of  coherent  signals  are  thus  linearly 
independent.  Therefore,  G  is  of  full  row  rank  and  the  modified  covariance  matrix  R$  is  of  full  rank. 
Otherwise,  if  K  <  max{Zi,  /2,  •  •  • ,  lg},  we  will  see  that  Rs  is  rank  deficient.  We  assume  that  Rsi  is 
the  correlation  matrix  associated  with  the  ith  group  of  coherent  signals.  Thus,  RSi  has  rank  1  and 
can  be  expressed  as  Ajhjh^  where  A i  and  h;  are  the  corresponding  eigenvalue  and  eigenvector  of 
Rsi-  We  have 


Rsi 

'  Axhxhf 

Rs  = 

Rs2 

= 

A2h2hf 

1 

Cr 

«Q 

ft? 

_ 1 

A9h9h^ 

(21) 


and 


E*r.iA1(X'tlh1)(C»,h,)Ir 

5  TtiMDk,  h2)(Dt,h2)H 

lbS  - 

ZLi^(Dkqhg)(Dkqhg)H 

(22) 

where  Dki  is  a  diagonal  matrix  consisting  of  li  diagonal  elements  of  Dk  which  are  associated  with 
all  the  DOA’s  from  ith  group  of  coherent  signals.  Since 


K 

Xi{Dkihi)(Dkihi)H)  <  min(K,li) 


k= 1 


(23) 


and 


K 

dim(^2  ^i(Dklhi)(Dkihi)H)  =  k  x  k, 


fc= 1 


(24) 


thus  Rs  is  rank  deficient  if  K  <  max{l\,l2i  •  •  • ,  lq). 

If  the  center  array  is  not  ambiguity  free,  then  all  the  b  vectors  associated  with  all  the  signals 
within  a  group  of  coherent  signals  can  be  linearly  dependent,  G  cannot  be  ensured  to  be  of  full  row 
rank,  and  neither  can  Rs. 
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Prom  Theorem  1  and  the  proof  above,  we  get  the  following  theorem. 


Theorem  2  SS  can  be  applied  to  an  array  of  arbitrary  geometry  to  obtain  a  full  rank  smoothed 
signal  covariance  matrix  if  and  only  if  an  array  has  an  orientational  invariance  structure,  its  center 
array  has  an  ambiguity  free  structure,  and  the  number  of  subarrays  is  larger  than  or  equal  to  the 
size  of  the  largest  group  of  coherent  signals. 

C  Further  Improvement 

To  get  a  smoothed  nonsingular  covariance  matrix  Rs  by  using  the  SS  technique,  we  need  K  > 
max{l\,l2,  ■  ■  ■ ,  lq}-  We  can  further  reduce  the  number  of  subarrays  by  getting  another  K  backward 
subarrays  similar  to  the  case  in  a  linear  array  [25].  Although,  the  Forward-Backward  Spatial 
Smoothing  (FBSS)  [25]  can  always  be  applied  in  a  uniformly  spaced  linear  array.  For  arrays  of 
arbitrary  geometry,  there  is  some  requirements  on  the  geometry  for  successful  implementation  of 
the  backward  method.  We  first  give  the  definition  of  central  symmetry: 

Definition  3  (Central  Symmetry)  The  array  is  central  symmetric  if  it  is  identical  before  and 
after  rotating  180°  about  its  center  of  mass. 

If  an  array  is  central  symmetric,  we  can  get  K  additional  backward  subarrays  by  reversing  the 
order  of  the  subarrays  and  the  order  of  the  sensors  within  each  subarray. 

Let  r bk(t)  denote  the  complex  conjugate  of  the  output  of  the  &th  backward  subarray  for  k  = 
1,  •  •  • ,  K.  We  have 

r  bk(t)  =  AlDk(DLs(t))*  +  n*(t)  (25) 

where  n(t)  is  an  additive  white  Gaussian  noise  vector,  Dl  is  a  diagonal  matrix  with  the  ith  diagonal 
element  given  by  e-j27r~rE  sm(/hcP-0i+f )  an(j  js  distance  between  the  first  sensor  in  the  first 
forward  subarray  and  the  first  sensor  in  the  first  backward  subarray.  The  angle  ^Kp  represents  the 
direction  of  the  line  on  which  the  two  sensors  are  located. 

The  covariance  matrix  of  the  fcth  backward  subarray  is  given  by 

Rbk  =  A1DkR»sD”A?  +  a2I  (26) 

with 

Rbs  =  E(D*Ls*(t)sT(t)Dl)  =  DIR*Dtl.  (27) 
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Define  the  spatially  smoothed  backward  subarray  covariance  matrix  Rb  as  the  average  of  these 
subarray  covariance  matrices,  i.e., 


K 


Rb  =  ^Y,Rk  =  MRbsA^  +  a2I, 


k=  1 


where 


Rbs  =  E  DkR\D», 


(28) 


(29) 


k= 1 


and  define  the  forward/backward  smoothed  covariance  matrix  R  as  the  average  of  R  in  (13)  and 
Rb,  i.e., 


R  =  =  AXR,A%  +  a2I. 


(30) 


It  follows  that 


Rs  = 


Rs  +  Rbs 


(31) 


We  can  show,  in  a  similar  way  as  in  the  case  of  a  linear  array  [25],  that  the  modified  source 
covariance  matrix  Rs  is  nonsingular  as  long  as  2 K  >  max{/i ,  1%,  •  •  • ,  lq}. 


Ill  Ambiguity  Free  Array  Structure 

To  perform  SS,  we  need  an  ambiguity  free  center  array  manifold.  Also,  to  perform  MUSIC,  we 
further  require  ambiguity  free  subarray  manifolds.  Ambiguity  arises  when  a  steering  vector  can 
be  expressed  as  a  linear  combination  of  other  steering  vectors  in  an  array  manifold  [4].  For  a 
uniformly  spaced  linear  array,  rank-1  ambiguity  [4]  cannot  be  avoided  since  the  DOA’s  which  are 
“mirror  images”  with  respect  to  the  array  line,  have  the  same  steering  vector.  This  limits  the  range 
of  DOA’s  estimable  by  a  uniformly  spaced  linear  array  to  within  180°.  Suppose  an  array  has  p 
elements,  then  rank-p  [4]  ambiguities  cannot  be  avoided.  In  this  paper,  an  ambiguity  free  array 
manifold  of  an  array  of  p  sensors  refers  to  rank-(p-l)  ambiguity  free.  Generally,  to  avoid  ambiguity, 
an  array  used  for  high-resolution  DOA  estimation  must  have  a  proper  structure.  An  ambiguity  free 
array  manifold  has  been  assumed  in  several  papers  [5],  [17],  [15].  Our  attempt  is  to  identify  all  the 
situations  in  which  ambiguity  may  arise.  One  of  our  guidelines  in  designing  arrays  is  to  avoid  these 
identified  ambiguities. 
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Theorem  3  In  an  azimuth  only  system,  the  necessary  and  sufficient  condition  for  an  ambiguity 
free  three-sensor  array  manifold  is  that  all  these  three  sensors  are  not  on  one  line  and  that  the 
distance  between  any  two  sensors  is  less  than  or  equal  to 

The  proof  is  given  in  the  appendix. 

We  can  see  in  general  that  (a)  rank-1  ambiguity  occurs  not  only  in  uniformly  spaced  linear 
arrays  but  also  in  rectangular  arrays  with  sensors  having  a  uniform  spacing  of  |  along  either  x-axis 
or  y-axis,  (b)  rank-2  ambiguity  occurs  in  an  array  that  consists  of  two  parallelly  positioned  linear 
arrays  with  an  identical  uniform  sensor  spacing  that  is  larger  than  (c)  rank-3  ambiguity  occurs  in 
an  array  that  consists  of  three  parallelly  positioned  linear  arrays  with  an  identical  uniform  sensors 
spacing  that  is  larger  than  and  (d)  higher  order  ambiguity  occurs  if  more  than  [|]  sensors  are 
on  one  line  in  a  A;  sensor  array  or  if  an  array  consists  of  m  parallelly  positioned  linear  arrays  with  an 
identical  uniform  sensor  spacing  that  is  larger  than  |  |_yj .  These  situations  are  shown  schematically 
in  Fig.3(a)-(d).  In  Fig.3  (b)  and  (c),  the  angles  Q  and  a  satisfy  the  following  constraint: 

2n^  sin(o:)  +  2kn  =  2nC^  sin(0),  ke{  1,2,  ••■}.  (32) 

A  A 

In  Fig.3  (d),  the  angles  6,  /3  and  a  satisfy  the  following  constraint: 

d  d  d 

27t—  sin(a:)  +  2/ci7t  =  27T--  sin(f3)  -I-  2&27T  =  27r^  sin(0),  (33) 

AAA 


where  kifa  €  {1, 2,  •  •  •}  and  k\  ^  fo. 

To  get  an  ambiguity  free  array  manifold,  it  is  necessary  to  avoid  these  identified  situations. 

IV  Implementation  and  Simulation  Results 

A  Some  Practical  Considerations 

To  determine  the  source  coherency  structure,  we  can  use  SRP  [30].  On  the  other  hand,  we  can 
estimate  the  maximum  number  of  incoming  angles  according  to  the  multipath  environment.  In  this 
paper,  limited  by  space,  we  assume  the  number  of  incoming  signals  is  given. 

In  practice,  we  can  perform  FBSS  by  setting  up  a  special  data  matrix.  Specifically,  for  the  nth 
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snapshot  we  set  up  the  data  matrix 


«(P,1,«) 

••  u(p,K,n) 

u*{l,K,n)  • 

'•  «*(l,l,n) 

AH(n)  = 

u(p-  1,1,  n)  • 

u(p  -  1  ,K,n) 

u*(2,K,  n)  ■ 

u*(2,l,n) 

.«(!,!,«) 

■■  u(l,K,n) 

u*(p,K,n )  • 

•  u*(p,l,n) 

where  u(i,j,n )  denotes  the  sample  taken  at  the  ith  sensor  of  the  jth  subarray.  For  the  totality  of 
N  snapshots,  we  can  define  the  overall  data  matrix 


Ah  = 


'ah(1),Ah(2),---,Ah(N)\. 


(35) 


It  follows  that  the  averaged  smoothed  correlation  matrix  R  (as  defined  in  (30))  can  be  estimated 
as  follows: 


R  = 


1 


2  NK 


AH  A. 


(36) 


As  we  know,  more  robust  results  can  be  obtained  from  data  domain  rather  than  from  covariance 
domain  [22].  We  can  proceed  with  MUSIC  [22]  algorithm  or  MVDR  [7]  [22]  beamforming  algorithm 
based  on  A  instead  of  R. 

An  array  needs  to  be  chosen  for  applying  SS.  Such  an  array  should  satisfy  all  conditions  afore¬ 
mentioned.  An  omnidirectional  circular  array  has  been  a  conventional  choice  for  mobile  commu¬ 
nications  [1]  [2],  and  there  have  been  active  research  efforts  to  find  a  pre-processing  scheme  for 
the  circular  array  to  handle  the  coherent  interference  [31].  However,  we  can  see  clearly  from  our 
discussion  that  a  single  circular  array  is  not  orientational  invariant.  Therefore  it  does  not  satisfy 
the  necessary  condition  for  applying  SS.  This  implies  that  the  circular  array  cannot  overcome  the 
coherent  interference  by  using  the  SS  technique.  For  some  circular  arrays  with  central  symmetric, 
we  can  apply  FBSS  to  handle  two  coherent  signals.  To  handle  more  than  two  coherent  signals, 
several  parallelly  positioned  circular  arrays  have  to  be  used. 


B  Selecting  Orientational  Invariance  Structure 

In  this  section,  we  study  some  guidelines  for  designing  an  optimal  sensor  array  for  SS.  We  found 
that  the  sensor  utilization  rate  is  an  important  factor  for  estimating  DOA’s  of  coherent  signals  with 
SS. 
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Definition  4  (Sensor  Utilization  Rate  (SUR)) 


(37) 

narray 

where  n subarray  is  the  sum  of  the  number  of  sensors  in  each  subarray,  and  narray  is  the  total 
number  of  sensors  in  the  whole  array.  Obviously  SUR  >  1,  because  of  possible  overlap  of  subarrays. 


SUR  = 


_  S  flsubarray 


Example  1:  We  perform  simulations  on  two  64-element  arrays:  (1)  a  hollow  square  array  as 
shown  in  Fig.4(a),  which  has  a  low  SUR  for  a  given  number  of  sensors,  and  (2)  a  dense  square 
array,  as  shown  in  Fig.4(b),  which  has  a  high  SUR.  The  dense  square  array  contains  4  subarrays 
each  having  49  sensors.  The  spacing  between  two  neighboring  sensors  is  0.45A.  The  SUR  of  the 
array  is  approximately  3.  The  hollow  square  array  contains  4  subarrays,  each  having  32  sensors. 
The  spacing  between  two  neighboring  sensors  is  0.45A.  The  SUR  is  2.  Both  structures  are  used  to 
estimate  the  DOA’s  of  two  coherent  signals.  The  input  SNR  is  20dB.  We  use  the  FBSS  method. 
The  simulation  results  are  shown  in  Fig.5.  In  cases  (a)  and  (b),  the  two  coherent  signals  are  at  40° 
and  50°.  Both  arrays  can  clearly  identify  the  DOA’s.  In  cases  (c)  and  (d),  the  two  coherent  signals 
are  at  45°  and  50°,  only  the  dense  square  array  can  determine  the  DOA’s.  Our  results  show  that 
a  dense  square  array  structure  is  better  than  a  hollow  square  array  structure. 

For  a  uniformly  spaced  linear  array,  the  larger  the  physical  aperture,  the  narrower  the  beamwidth, 
and  the  higher  the  resolution.  For  a  nonlinear  array,  the  DOA  resolution  is  however  decided  by  an 
effective  array  beamwidth.  The  beamwidth  b\^  of  two  signals,  with  DOA  at  0\  and  02,  arriving  at 
an  array  with  steering  vector  v(0)  is  evaluated  by  the  following  equation. 


lv(fli)Hv(fl2)L 

!v(#i)I|v(02)| 


(38) 


We  compared  the  beamwidths  between  a  dense  square  subarray  and  a  hollow  square  subarray.  In 
Fig.  6,  we  plot  the  beamwidths  of  two  signals,  that  are  10°  and  5°  apart  in  (a)  and  (b),  respectively, 
versus  the  angle  of  arrival  of  the  signal  that  has  a  smaller  DOA  values.  We  see  that  although  the 
dense  square  array  has  a  smaller  physical  array  aperture,  it  has  a  narrower  beamwidth.  Since  both 
arrays  have  the  same  number  of  elements,  we  infer  that  SUR  is  an  important  factor  and  needs  to  be 
maximized  in  the  array  design.  An  array  consisting  of  several  parallelly  positioned  circular  arrays 
has  a  low  SUR  and  is  not  recommended  for  SS  applications. 


14 


C  Simulation  Results 


In  this  section,  we  present  some  simulation  results  on  MUSIC  algorithm  to  show  the  effective¬ 
ness  and  applications  of  our  SS  and  FBSS.  We  choose  a  square  array,  which  has  an  orientational 
invariance  structure,  central  symmetric,  and  a  sensor  spacing  less  than  |. 

Example  2:  To  compare  the  performance  of  a  square  array  to  that  of  a  linear  array  under 
the  same  complexity,  we  use  a  nine-sensor  linear  array  and  a  nine-sensor  square  array  as  shown  in 
Fig.7,  both  having  a  spacing  of  0.45A  between  neighboring  sensors.  We  divide  the  both  arrays  into 
four  overlapped  subarrays.  We  get  six  and  four  sensors  in  each  subarray  of  the  linear  array  and 
the  square  array  respectively.  The  resolution  of  DOA  estimations  is  decided  by  the  beamwidth  of 
the  subarrays.  We  consider  two  narrow-band  coherent  signals  with  DOA’s  at  70°  and  85°.  The 
SNR  is  20  dB.  A  total  of  500  samples  (“snapshots”)  are  taken  from  the  array.  We  use  SS  as  a 
pre-processing  scheme  for  MUSIC.  Fig.8  shows  that  the  DOA’s  of  the  two  coherent  signals  are  not 
resolved  using  a  linear  array,  whereas  the  square  array  gives  a  satisfactory  result.  In  Fig.  9,  we  plot 
the  beamwidths  of  two  signals,  that  are  15°  apart,  versus  the  angle  of  arrival  of  the  signal  that  has 
a  smaller  DOA  values.  For  the  linear  array,  actual  beamwidth  varies  greatly  with  the  DOA.  The 
square  array,  however,  shows  a  relatively  stable  beamwidth  for  different  DOA’s.  When  DOA’s  are 
at  70°  and  85°,  the  square  array  provides  a  narrower  beamwidth  and  a  higher  resolution. 

Example  3:  We  use  the  square  array  shown  in  Fig.7  to  receive  two  coherent  signals  with 
DOA’s  at  75°  and  100°.  We  apply  SS  and  FBSS  separately.  Fig.  10  shows  that  the  DOA  estimation 
resolution  achieved  by  a  central  symmetric  array  is  significantly  improved  by  using  the  FBSS 
method. 

Example  4:  In  a  wireless  fading  channel,  coherent  signals  can  not  be  avoided.  Fig.  11  shows  a 
typical  example  of  the  distribution  of  scatters  in  a  mobile  radio  environment  [33] .  We  can  treat  all 
the  reflected  signals  from  local  scatterers  as  coming  from  a  super  position.  However,  those  reflected 
signals  from  far  away  scatterers  such  as  high  rise  buildings  or  mountains  are  all  coherent  signals. 
We  use  a  dense  square  array  of  sixty-four  sensors  as  shown  in  Fig.4(b)  as  the  base  station  antenna. 
The  array  contains  4  subarrays  each  of  49  sensors.  The  spacing  between  two  neighboring  sensors 
is  0.45A.  The  array  receives  signals  from  four  mobile  users:  four  coherent  signals  at  20°,  65°,  150° 
and  200°  from  user  1,  three  at  230°,  250°  and  280°  from  user  2,  two  at  30°  and  300°  from  user 
3  and  one  at  320°  from  user  4.  The  SNR  is  20  dB.  A  total  of  500  samples  are  used.  First,  we 
apply  FBSS  and  then  apply  MUSIC.  Simulation  results  are  shown  in  Fig.12.  According  to  these 
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DOA’s  information,  we  can  further  perform  constrained  beamforming  and  thus  can  achieve  Spatial 
Division  Multiple  Access(SDMA)  [34]  in  a  multipath  environment. 

We  found  the  results  obtained  for  a  nonlinear  array  in  an  azimuth-only  system  remain  valid  in 
an  azimuth-elevation  system.  The  following  is  an  example. 

Example  5:  We  use  the  square  array  shown  in  Fig. 7  to  receive  two  coherent  signals,  one  is  at 
an  azimuth  of  40°  and  an  elevation  of  30°,  and  the  other  is  at  an  azimuth  of  50°  and  an  elevation  of 
60°.  The  SNR  is  20dB.  The  number  of  samples  taken  is  500.  By  using  FBSS  and  MUSIC,  we  obtain 
the  result  in  Fig.13.  It  demonstrates  that  a  planar  array  enables  us  to  perform  DOA’s  estimation 
in  a  3D  domain  while  a  linear  array  is  not  capable  of  doing  that. 


V  Spatial  Smoothing  for  ESPRIT 


Similar  to  MUSIC,  the  ESPRIT  algorithm  [5]  is  an  approach  to  signal  parameter  estimation.  It 
exploits  an  underlying  data  model  at  significant  computational  savings.  The  ESPRIT  algorithm  is 
also  limited  to  estimating  parameters  in  noncoherent  incoming  signals.  The  conventional  SS  can 
be  incorporated  into  ESPRIT  [32],  but  it  requires  the  center  array  to  be  a  uniformly  spaced  linear 
array.  In  this  section,  we  show  that  our  scheme  also  works  for  the  ESPRIT  algorithm  to  estimate 
parameters  in  a  coherent  interference  environment. 

In  the  ESPRIT  algorithm,  we  consider  d  narrow-band  plane  waves  with  incident  angles  9i,  -  ■  -Od, 
and  wavelength  A,  impinge  on  a  planar  array  of  m  sensors  (m  is  even),  arranged  in  y  doublet  pairs. 
The  displacement  vector  is  the  same  for  each  doublet  pair,  but  the  location  of  each  pair  is  arbitrary. 
The  sensor  output  x(t)  is  given  by 


x(t)  = 


s(t)  +  n(t) 


(39) 


where  n(t)  is  a  white  Gaussian  noise  vector.  A  and  A<f>  are  the  steering  matrices  corresponding  to 
the  first  sensors  and  the  second  sensors  in  all  pairs,  respectively.  The  matrix  $  is  a  diagonal  d  x  d 
matrix  of  phase  delays  between  the  doublet  sensors  for  the  d  signals.  The  sensor  output  covariance 
matrix  Rx  is  thus  measured  by 


(  A  ) 

(  A  \ 

Rx  — 

R, 

{A*J 

l-4®) 

(40) 
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A  full  rank  matrix  Rs  is  assumed  when  the  ESPRIT  algorithm  is  performed.  If  some  of  the 
incoming  signals  are  coherent,  Rs  will  not  be  a  full  rank  matrix  and  the  ESPRIT  will  fail.  The 
spatial  smoothing  technique  we  introduced  in  the  previous  sections  can  then  be  applied  here  to  get 
a  modified  full  rank  signal  covariance  matrix. 

We  consider  each  doublet  sensor  pair  in  the  array  used  by  ESPRIT  algorithm  as  one  element. 
Then  the  array  consists  y  elements.  If  this  array  has  an  orientational  invariance  structure  with  K 
subarrays  and  the  corresponding  center  array  has  an  ambiguity  free  structure,  the  sensor  output 
at  the  /c th  subarray  is  given  by 


Xfe(t)  = 


'  M  \ 


Dks(t)  +  n(t). 


(41) 


Matrix  D k  is  a  diagonal  dxd  matrix  of  the  phase  delays  in  the  form  given  in  (11).  The  corresponding 
covariance  matrix  RXk  is  given  by 


R 


Xk  — 


Ax 


/ 

DkRsDk 

\ 


Ax 


(42) 


A  smoothed  output  covariance  matrix  Rx  can  thus  be  defined  as 

1  K 

=  7F  ^2  Rxk  = 


K  f 

i=i 


A,  ) 

Al  ) 

Rs 

y  J 

+  a2I 


(43) 


where  Rs  is  the  modified  signal  covariance  matrix  as  defined  in  (14).  As  proved  in  Section  II,  Rs 
is  of  full  rank  if  K  is  larger  than  or  equal  to  the  size  of  the  largest  group  of  coherent  signals.  We 
can  now  successfully  perform  ESPRIT  based  on  Rx.  We  can  also  use  FBSS  to  further  reduce  the 
number  of  sensors  required  and  to  improve  the  estimation  resolution  if  the  array  of  y  element  is 
central  symmetric. 

Although  SS  enables  ESPRIT  to  estimate  DOA’s  in  a  coherent  interference  environment,  the 
estimation  is  still  limited  to  identifying  DOA’s  within  180°  in  an  azimuth  only  system.  Hence,  in 
terms  of  performance  robustness  to  DOA’s,  our  SS  is  more  effective  for  MUSIC  than  for  ESPRIT. 

Example  6:  A  twelve  sensor  array  shown  in  Fig.  14  is  used  in  this  example  to  receive  two 
coherent  signals  at  70°  and  80°.  This  array  consists  of  two  overlapping  nine-sensor  square  arrays. 
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Each  sensor  in  one  square  array  and  its  counterpart  in  another  form  a  doublet  pair.  These  nine 
doublet  pairs  form  an  array  which  has  orientational  invariance  structure  and  is  central  symmetric. 
The  spacing  between  two  neighboring  sensors  is  0.45A.  The  doublet  spacing  for  ESPRIR  is  0.45A. 
The  SNR  is  20dB.  A  total  of  2000  trials  are  run.  A  histogram  of  the  results  is  given  in  Fig.15.  We 
apply  FBSS  first  and  then  applied  the  ESPRIT.  The  two  angles  are  clearly  identified. 

VI  Conclusions 

To  significantly  improve  performance  robustness  in  DOA  estimation  and  in  adaptive  beamforming, 
we  developed  techniques  for  applying  SS  on  arrays  of  nonlinear  geometry,  thus  making  MUSIC, 
ESPRIT  and  adaptive  beamformers  operative  in  a  coherent  interference  environment.  In  order 
to  apply  SS  to  an  array  of  nonlinear  geometry,  this  array  must  have  an  orientational  invariance 
structure  and  its  center  array  must  be  ambiguity  free.  Also  the  number  of  subarrays  must  be  greater 
than  or  equal  to  the  largest  number  of  mutually  coherent  signals.  To  apply  SS  in  conjunction 
with  MUSIC,  all  the  subarrays  must  also  be  ambiguity  free,  and  the  number  of  sensors  in  each 
subarrays  must  be  larger  than  the  number  of  incoming  signals.  For  ESPRIT,  two  identical  arrays  (or 
subarrays)  separated  by  a  displacement  vector  are  used  each  satisfying  the  conditions  for  applying 
SS  and  MUSIC. 

When  a  nonlinear  array  is  central  symmetric,  the  FBSS  can  be  used  and  it  outperforms  the 
regular  SS  in  terms  of  improved  efficiency  and  estimation  resolution. 

We  proved  the  necessary  and  sufficient  conditions  for  a  three-sensor  array  manifold  to  be  am¬ 
biguity  free.  We  identified  several  situations,  for  higher  order  sensor  array  manifolds,  in  which 
ambiguity  may  arise.  It  is  necessary  to  avoid  the  identified  ambiguities  in  designing  ambiguity  free 
center  arrays  and  subarrays. 

In  practice,  we  found  that  we  can  choose  a  square  array  with  a  sensor  spacing  less  than  | 
to  meet  all  the  conditions  required  for  applying  SS.  Simulation  results  also  show  that  for  DOA 
estimation  of  coherent  signals  using  SS,  a  square  array  has  a  prefered  geometry  in  terms  of  the 
DOA  estimation  resolution  and  performance  robustness. 

VII  Appendix 

Proof  of  Lemma  1: 
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If  part: 


The  proof  is  obvious  and  is  omited. 

Only  If  part: 

If  B  —  AC,  A  —  [a{0\),  •  •  • ,  a(0d)]  and  B  =  [6(0i),  •  •  • ,  6(0^)]  and  also  assume  C  is  not  a  diagonal 
matrix,  i.e.  it  has  non-zero  element  for  l  /  m,  then  the  steering  vector  b(6m)  is 

d  d 

b{@m)  —  ^  =  "t"  ^  ^  •  (44) 

t= 1  i=\,i^Ll 

This  means  that  b(0m)  is  a  function  of  variable  0/,  which  contradicts  to  the  definition  that  b(()rn)  is 
only  a  function  of  0m.  Thus  the  assumption  that  C  is  a  non-diagonal  matrix  is  false.  C  has  to  be 
a  diagonal  matrix  with  ca  = 

Proof  of  Lemma  2: 

If  part: 

Obviously,  B  can  be  any  of  {Ai,  A2,  ■■■ ,  Ak}- 

Only  if  part: 

If  each  A{  can  be  mapped  to  a  steering  matrix  B,  by  definition  there  exist  C\,  Cj  such  that 
Ai  =  BCi,  Aj  —  BCj.  By  Lemma  1,  C*  is  a  diagonal  matrix.  So  Cf1  exists  and  is  also  a  diagonal 
matrix.  We  have  Aj  =  AlCf'CJ.  Let  C„  =  C~lCj,  c„  is  the  product  of  two  diagonal  matrices. 
So  Cij  is  also  an  diagonal  matrix.  Aj  =  AiCij. 

Proof  of  Theorem  3: 

If  part: 

If  sensors  A,  B  and  C  are  not  on  one  line  and  their  mutual  distance  is  less  than  | ,  without  loss 
of  generality,  we  let  sensor  A  be  the  first  sensor  in  the  array,  B  the  second  and  C  the  third.  The 
steering  matrix  of  the  array  has  the  form 


1 

1 

1 

>3<t>  l(0l) 

ej<Pi{62) 

ej<p2(d2) 

ej<p2((>3) 

(45) 


where  4>  denotes  phase  delay.  If  the  distance  between  any  two  sensors  is  <  |,  the  phase  delay  0i(0j) 
and  4>2(0i)A  =  1, 2,  3,  are  real  numbers  from  (— 7r,7r). 

Note  that  the  steering  matrix  of  the  array  corresponding  to  three  incoming  signals  at  different 


19 


angles  is  a  special  case  of  the  general  array  in  Lemma  2  of  [28].  By  Lemma  2  in  [28],  V  is  nonsingular 
with  possible  exception  in  one  of  the  following  three  situations: 

(1)  When  0i(#i)=</>i(02)5  be.  the  two  incoming  signals  are  symmetric  with  respect  to  the  line 
on  which  sensors  A  and  B  are  located.  Note  that 


det(V)  = 


0  1  0 

ej<Pi(6i)  _  ej<t>  1(62)  ej</>  1(62)  ej<t>i((h)  _  ej<Pi(02) 
ej<t>2(.6l)  _  ej<j> 2(02)  ej<t> 2(62)  ej<j> 2(03)  _  ej<p2(02) 


ei4>l(6l)  _  ej<t>  1(82)  ej<t>l(03)  _  ej<f>  1(62) 

ei<t> 2(81)  _  gjfofa)  ej<t>2(83)  _  ej(t>2(82) 


(46) 


(47) 


When  (pi (6i)  =  ^1(62),  det(F)  =  0  if  and  only  if  ^2(^1)  =  ^2(^2)  or  $1(63)  =  $1(62).  Since  these 
sensors  are  not  on  one  line,  if  (pi{9\)  =  <p  1(^2),  we  have  02(0i)  ^  02(02)-  Since  0\ ,  9  2  and  63 
are  three  different  angles,  when  9\  and  O2  are  symmetric  with  respect  to  the  line,  63  and  62  can 
not  be  symmetric  to  the  line,  i.e.  if  (pi{9{)  =  0i(f?2),  then  we  get  0i(03)  ^  0i(02)-  Thus,  when 
<pi  (0i )  =  0i(02)>  the  matrix  V  is  nonsingular. 

(2)  Similarly,  we  can  prove  that  when  02 (#i)  =  <£2 (#2),  the  matrix  V  is  nonsingular. 

(3)  When  0i(#i)  -  <j)\ (62)  =  02(0i)  -  hi^),  be.  0i(0i)  -  02(0i)  =  0i(#2)  -  ^>2(^2),  <?i  and  02 
are  symmetric  with  respect  to  the  line  connecting  sensors  B  and  C.  Note  that 


det(V) 


e~j<t>2(0l)  e-j<f>2(02)  e~j<l>2(03) 

ej<t>l(8l)~j<t>2(0l)  ej<!>l(82)-j4>2(02)  ej4>l(03)-j<p2{83) 

1  1  1 

(e-j<t>2(dl)  _  e-j<t>2(82)^ej(l>l(82)-j<p2(02)  _  ej<t>\{.83)-j(j)2{83)^ 


(48) 

(49) 


When  4>i(9i)  —  cp 2(^1)  =  0i(02 )  —  02(#2),  det(V)  =  0  if  and  only  if  e  —  e  j<t> 2(82)  or 

ei4>\(82)-j<t>2{di)  _  ej4>A8z)-j<}>2{83)'  gjnce  mutual  distance  between  A,  B  and  C  are  less  than 
3,  02(^i),  02(#2))  <£i (#2)  ~~  02(^2)  and  0i(03)  -  02(#3)  are  all  real  numbers  in  (-7r,7r).  e ~iM8i)  — 
e~j<h(02)  jf  anrj  only  if  02 (#1)  =  02 (6*2) .  ei<t>d^2)-j4>2{82)  —  ej  4>i  (83) -j  42(83)  jf  an(j  only  if  0i(^2)  — 

02(^2)  =  01  (#3)  -  02(^3)- 

Since  A,  B  and  C  are  not  on  one  line,  if  6\  and  62  are  symmetric  to  the  line  connecting  B  and 
C ,  they  can  not  be  symmetric  to  the  line  connecting  A  and  B  or  A  and  C.  i.e.,  if  0i(#i)  —  02 (#1 )  = 
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01  (02)  ~  02(02),  we  have  0 2(^1)  7^  02(02)-  Since  0i,  02  and  63  are  three  different  incoming  angles, 
if  0i  and  02  are  symmetric  to  the  line  connecting  5  and  C,  02  and  03  can  not  be  symmetric  to  the 
line,  i.e.,  if  0i(0i)  -  02 (0i)  =  0i (^2)  -  02(02),  we  have  0i(02)  -  02(02)  /  0i(03)  -  02(03)-  Thus, 
when  0i  (0i )  —  02(0i)  =  0i  (02)  —  02(02),  the  matrix  V  is  nonsingular. 

Therefore,  we  conclude  that  all  the  three  situations  which  cause  the  singularity  of  the  matrix 
in  Lemma  2  of  [28]  will  not  cause  the  singularity  of  three-sensor  steering  matrix  if  three  sensors 
are  not  on  one  line  and  their  mutual  distance  is  less  than  Therefore  the  matrix  V  is  full  rank. 

If  the  spacing  between  any  two  of  the  three  sensors  is  not  larger  than  |,  and  there  is  at  least 
one  pair  in  these  three  sensors  with  a  spacing  of  then  the  only  situation  that  the  phase  delay 
01  (0j)  and  02(0j),i  =  1,2,3,  are  not  all  in  ( — 7r, 7r)  is  when  one  of  the  incoming  signals  is  from 
the  direction  parallel  to  a  line  on  which  the  two  sensors  with  spacing  |  are  located.  The  other 
two  signals  can  be  either  from  the  opposite  direction  or  from  other  directions.  If  one  of  the  other 
two  signals  is  from  the  opposite  direction,  it  can  be  easily  proved  that  the  corresponding  steering 
matrix  is  full  rank.  If  the  other  two  signals  are  from  the  two  other  different  directions,  then  one  of 
0n(0j),  n  =  1,2,  i  =  1,2,3  is  equal  to  n  and  the  rest  are  real  numbers  from  (— 7r,7r).  Similarly,  we 
can  prove  that  the  matrix  V  is  of  full  rank. 

Only  if  part: 

If  the  conditions  in  Theorem  1  are  not  satisfied,  rank-1  or  rank-2  ambiguity  occurs  for  some 
incoming  signals.  These  situations  are  shown  schematically  in  Fig.2(a)(b).  In  Fig.2(a),  the  relation 
between  0  and  a  is 


d>  d 

2n — sin(9  —  a.)  +  k2n  =  2n — sin(0  +  a)  k  G  {1,2,  •  •  •}.  (50) 

A  A 

In  Fig.2(b),  the  relation  between  0  and  a  is 

2Tr^sin(a)  +  k2-K  —  2-K^sin(^- —  6),  k  E  {1,2,  ••  •}.  (51) 

A  A  Z 
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Figure  1:  Orientational  invariance  sensor  array  geometry 


(a)  rank-1  ambiguity 


(b)  rank-2  ambiguity 

Figure  2:  Three-sensor  array  structures  that  can  cause  ambiguities 
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(a)  rank-1  ambiguity 


(b)  rank-2  ambiguity 


(c)  rank-3  ambiguity 


(d)  high  order  ambiguity 


Figure  3:  high  order  array  structures  that  can  cause  ambiguities 
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Figure  4:  (a)  A  sixty  four-sensor  dense  square  array  with  four  overlapping  dense  square  subarrays 
of  forty  nine-sensors  (b)A  sixty  four-sensor  hollow  square  array  with  four  overlapping  hollow  square 
subarrays  of  thirty  two-sensors 


(C)OOA  at  46,80  (<J)DOA  at  45.80 


Figure  5:  A  sixty  four-sensor  hollow  square  array  is  used  in  (a)  and  (c),  A  sixty  four-sensor  dense 
square  array  is  used  in  (b)  and  (d) 
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Figure  6:  Beamwidth  comparison  between  a  dense  square  array  and  a  hollow  square  array  (a)  two 
DOAs  are  10°  apart  (b)  two  DOAs  are  5°  apart 
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Figure  7:  A  nine-sensor  square  array  with  spacing  d 


Figure  8:  SS  and  MUSIC  for  DOA  estimation  of  two  coherent  signals  at  70°  and  85° 


Figure  9:  Beamwidth  comparison  between  a  six-sensor  linear  array  and  a  four-sensor  square  array 


Figure  10:  Standard  deviation  of  DOA  estimation  using  SS  and  FBSS 


Figure  11:  Multiuser  Frequency  Selective  Channel 


Figure  12:  DOA  estimation  of  four  groups  of  coherent  signals  at  (20°, 65°, 150°, 200°), 
(230°, 250°, 280°),  (30°, 300°)  and  320°  based  on  a  sixty  four-sensor  square  array 
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Figure  13:  DOA  estimation  of  two  coherent  signals  at  an  azimuth  of  40°  and  an  elevation  of  30°, 
and  at  an  azimuth  of  50°  and  an  elevation  of  60°,  respectively 


|,  d  ,| 

Figure  14:  A  twelve-sensor  rectangle  array  with  spacing  d, 


Figure  15:  FBSS  and  ESPRIT  for  DOA  estimation  of  two  coherent  signals  at  70°  and  80° 
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